METHOD AND DEVICE FOR VISUALIZING COMPUTER- GENERATED 
INFORMATIONS 

FIELD OF THE INVENTION 

The invention relates to a method and a device for visuali- 
zing computer-generated informations. 

BACKGROUND OF THE INVENTION 

WO 00/5253 6 discloses displaying computer-generated infor- 
mations, such as in particular data bank and state informa- 
tions concerning technical parts together with a real image 
of the corresponding technical part on a display, especial- 
ly in text form. The joint display of real images and com- 
puter-generated informations is called augmented reality 
(AR) . 

In earlier times industrial robots were usually programmed 
once and for all at the time of putting into operation and 
then carried out for many years the same task without any 
change. Within the framework of more flexible automation 
reprogramming is more important as a time and cost factor. 

In order to move a robot into a predetermined position or 
out of the way, it can be moved manually, particularly by a 
user using movement keys or also controlled by means of a 
6D mouse. The robot movement direction is dependent on the 
selected reference coordinate system. It is not always im- 
mediately clear to the user which reference coordinate sy- 
stem has been selected or how this is oriented in space. In 
practice, careful trial and error is frequently used to 
establish whether the robot is moving in the desired direc- 
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tion. An error can lead to serious damage or even injuries 
to persons . 

To a particular extent this applies to users having little 
robot experience, because the conception of reference coor- 
dinate systems in the case of problems assumes knowledge 
which a robot user must possess in order to be able to ope- 
rate effectively with a robot system. However, it is often 
difficult to make this comprehensible to "robot movices", 
because they do not generally have the corresponding mathe- 
matical qualifications . 

Nowadays the measurement of the tool centre point (TCP) on 
an industrial robot usually takes place manually. Thus, 
e.g. a preset point is moved up to a number of times from 
different directions using the tool centre and the cor- 
responding robot positions are stored and from these the 
position and optionally the orientation of the tool centre 
with respect to the robot hand flange is determined. The 
measurement of workpieces, i.e. the establishing of basic 
coordinate systems takes place in a similar way. 

As a function of the intended use of the robot system 
(nature of the tool - welding, cutting, adhering and simi- 
lar tool and its orientation with respect to the workpie- 
ce) , the robot user must select a specific measurement me- 
thod. If e.g. the zero point of the workpiece coordinate 
system to be measured is located outside the operating area 
of the robot, measurement has to take place indirectly 
using known points in space. A different measurement me- 
thod must be chosen if the workpiece to be machined is gui- 
ded on a gripper fixed to the robot flange. In order to 
cover all use cases more than 18 different measurement me- 
thods have become known. Even after comprehensive trai- 
ning, a robot user is generally unclear as to which measu- 
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rement method should be used in a specific case. Even ex- 
perienced users come up against knowledge deficiencies when 
new configurations occur, such as other "tools" and/or 
"workpieces " , particularly as measurements are rarely per- 
formed . 

The measurement procedure can be very complex and involves 
a considerable amount of time. In the frequently occurring 
case of a workpiece guided by a robot and with the tool 
stationary, it is e.g. necessary to carry out a triple mea- 
surement of positions and then possibly a determination of 
the orientation. A careful procedure is necessary, because 
an imprecise positioning can lead to a measurement method 
yielding no or no reliable results. The lack of visibility 
of the measurement results also leads to uncertainty as to 
whether a good measurement result exists or whether it has 
been falsified by method errors, incorrect inputs, etc. 

Robot programs are nowadays frequently produced by tea- 
ching, in that the robot is moved manually to the target 
position of a movement and the corresponding path point is 
then stored. The path points of programs produced offline 
by means of a simulation must frequently subsequently be 
taught in situ, because the geometry of the simulation mo- 
del and the real robot use environment seldom precisely 
coincide to an adequate extent. Even when replacing a de- 
fective robot mechanism by another such mechanism the path 
points in a robot program frequently have to be taught 
again, because the new mechanism has a slightly differing 
geometry as a result of manufacturing tolerances. 

As robots may only move very slowly when humans are in the 
vicinity and the target position must be moved up to preci- 
sely, teaching or af terteaching is time-consuming. Moreo- 
ver, despite the slow movement there is always a certain 
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risk of damage to things and injury to people. This risk 
more particularly exists in the case of newly produced ro- 
bot programs (e.g. offline by means of a simulation) , which 
the programmer must allow to run on the robot for testing 
purposes . 

When a problem or fault occurs in operation, the robot is 
frequently moved back along the programmed path. In such 
or other complex situations, the behaviour of the robot sy- 
stem cannot be directly foreseen by the user. Here again 
damage to things and injury to people can occur. 

In order to exclude collisions, for robot systems an opera- 
ting area is fixed and which may not be penetrated by the 
robot tool or the entire robot or which the robot tool or 
robot may not leave. Other operating areas are jointly 
used by several "cooperating" robot systems and/or other 
equipment components and/or humans, so that the use thereof 
must be managed. 

The fixing of operating areas in such an installation pre- 
supposes complicated measurement. The monitoring or adap- 
tation of fixed operating areas in the plant is also very 
complicated, because no optical representation or display 
exists. It is unclear in the plant whether and in which 
zones the attainable operating areas of robot systems over- 
lap, so that a collision risk exists. 

In the case of manually guided operation of robot systems 
(robots as an intelligent assist device or "cobot" ) the de- 
grees of freedom of motion of the robot are often artifici- 
ally restricted in order to facilitate the activities of 
the user. Through virtual fixtures the robot effector is 
e.g. control-guided, e.g. always within one plane, in which 
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the orientation is always kept parallel to the ground or 
the like. 

Virtual fixtures constitute a special case of restricting 
the operating area of a robot system. Operating areas are 
volumes which are not left by the robot system or part the- 
reof or which must not be penetrated by the same. In spe- 
cial applications in which a robot is guided by the user 
with respect to the tool, it can be appropriate to fix the 
orientation of the gripper on a software basis, whereas the 
gripper position in space can always be freely changed by 
the user. One application is the use of a robot as a joi- 
ning aid in assembly. The robot carries the possibly heavy 
and bulky component and the user can move it with limited 
force expenditure. From the outset the component is in the 
position in which it is to be assembled. Another applica- 
tion is e.g. the painting of surfaces, in which the robot 
holds the tool (i.e. the paint spray gun) in a predetermi- 
ned distance from the surface to be painted, whilst the 
user guides the gun over the surface. Such planned soft- 
ware-based restrictions to the movement possibilities are 
referred to as virtual fixtures, because they act in a man- 
ner similar to a mechanical forced guidance of the robot 
gripper . 

The information concerning the operating areas or virtual 
fixtures are present as abstract mathematical information 
in the robot control . 

The virtual fixture can change dynamically as a function of 
the present task. Fixing or modifying virtual fixtures or 
operating areas during programming exceeds the necessary 
conception capacity of many users, so that it is not imme- 
diately apparent to the user in which directions a path 
and/or rotary movement of the robot or more precisely its 
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tool centre point or TCP is possible or is blocked. As he 
cannot foresee the behaviour of the robot system, he is ma- 
de uncertain. As a result of incorrect manipulation, such 
as e.g. the exerting of high forces in one direction, which 
are blocked by a virtual fixture, damages can arise on the 
robot system. 

In present manufacturing plants frequently numerous robots 
are tightly juxtaposed. Each of these robots has its own 
manual programmer. A clear association between the manual 
programmer and the robot in principle exists by means of an 
optionally present connecting cable between the two. In 
this situation the subsequent tracking of one specific ma- 
nual programmer with the associated robot connecting cable 
during the putting into operation and programming of a ro- 
bot system and also possibly during subsequent operation 
takes much effort and time, so that in practice use is ge- 
nerally made of the trial and error method, in which a ma- 
nual programmer is carefully operated and a check is made 
as to whether the desired robot moves and if not another 
manual programmer is tried out. The same applies if no ca- 
ble is present and instead communication between the manual 
programmer and the robot control takes place in wireless 
form. 

The problem of the invention is to provide a method and de- 
vice by means of which there is a simplification to the 
operation of one or more robots for the purpose of setting 
up, programming, teaching of the movement sequences to be 
performed by them and the like. 

SUMMARY OF THE INVENTION 

In the case of a method of the aforementioned type, the 
problem of the invention is solved in that a determination 
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of the position and orientation or pose of the image recei- 
ving device takes place and that robot-specific informati- 
ons corresponding to this determination are faded over the 
image of the real environment on the viewing device. 

With regards to the device, the problem is solved by a de- 
vice for the visual fading of computer-generated informati- 
ons into an image of the real environment, in which there 
is a means for determining the pose of the image receiving 
device and a means for fading robot-specific informations 
corresponding to the determination over the image of the 
real environment on the viewing device. 

Within the scope of the present invention the term robot- 
specific information comprises information relating to the 
operation of a robot, such as particularly information con- 
cerning its operating and program data, on the actual robot 
itself, the workpieces to be manipulated by the robot and 
obstacles located in its vicinity such as equipment, parts, 
in each case concerning the pose and movement. The infor- 
mation is in particular faded in pictorially onto the vie- 
wing device. 

The invention permits greater flexibility of the automation 
of operating sequences by means of robots, because repro- 
gramming is facilitated. 

The invention makes the putting into operation, programming 
and operating of robots more simple and efficient. The in- 
vention more particularly relates to groups of robots of 
complex installations having additional robot components, 
such as conveying installations. 

According to a first, preferred development of the inventi- 
on, at least one robot-specific reference coordinate system 
is faded in and in particular hand flange- fixed coordinate 
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systems are faded in. The visualizing of the robot hand 
flange coordinate system and therefore also the tool fixed 
to the hand preferably takes place in such a way that the 
origin of this coordinate system is preferably reproduced 
on the TCP (TCP coordinate system) . It is also possible to 
visualize other reference coordinate systems, such as in 
particular the stationary "world coordinate system" or sta- 
tionary workpiece-f ixed, basic coordinate system, the lat- 
ter preferably being in the centre of symmetry of the sta- 
tionary robot base. 

In the case of static basic coordinate systems (world, ba- 
se) as a function of the given requirements, additionally 
or alternatively it is possible to fade in a coordinate sy- 
stem shifted into the TCP and whose directions in space 
correspond to that of the stationary reference coordinate 
system, but which moves jointly with the robot flange. 

According to another preferred development, the robot axes 
are faded in. The virtual robot axes are so superimposed 
on the real image of the robot, that they correspond to the 
axes of the individual parts of the robot (base, rocker, 
robot arm and robot hand) or coincide therewith and in this 
way represent the same. 

The coordinate directions are faded in on the robot system, 
e.g. on its tool centre point and optionally also designa- 
ted (e.g. X, Y, Z in the case of the cartesian method and 
e.g. axis 1, axis 2, axis 3, etc. for the axis-specific me- 
thod) . It can also be appropriate to accentuate the active 
coordinate direction or directions or axis or axes graphi- 
cally, e.g. by colour or animation. 

In order to permit a direct manipulation of reference coor- 
dinate systems a coordinate system of the selected type is 
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automatically generated at a predetermined position and vi- 
sualized. The coordinate system can be shifted and turned 
by the user under visual control until the desired position 
and orientation or pose is reached. 

The inventive visualization of the selected reference coor- 
dinate system or robot axes on the real robot system by 
means of augmented reality (AR) , makes it more easily for 
the user to foresee the manual displacement, particularly 
by means of the robot system displacement keys. 

The manipulatability of the reference coordinate systems 
made possible by the invention makes it unnecessary for the 
user to learn and carry out cumbersome, complicated measu- 
rement methods. A manipulation of augmented, virtual ob- 
jects is intuitive, saves time and cost and only requires 
reduced qualifications on the part of the user. 

Another preferred development of the invention is characte- 
rized in that the image of a control element of a robot ma- 
nual programmer movable in at least two dimensions is faded 
in and in particular the image of the control element is 
faded in in association and orientation of the robot hand 
flange. It is made clear to the user and conception is fa- 
cilitated as to which movement the robot and in particular 
the hand thereof, together with a tool fixed thereto will 
be carried out if the real control element, e.g. on a manu- 
al programmer is operated. The control element can either 
be faded in on the robot system TCP and moved therewith, or 
the input element can be faded onto the display at a prede- 
termined position, or can be fixed in space. 

In the case of control elements having a symmetrical con- 
struction, the position of the coordinate system in the fa- 
ded-in input element may not be directly detectable for the 
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user. In such a case it can be appropriate to make addi- 
tional markings, (e.g. a graphic symbol) on the faded- in 
input element. In this sense e.g. also the normal position 
of the hand of the user on the control element can be faded 
in. With a control element having six degrees of freedom 
(e.g. a 6D mouse) , it may also be sufficient to fade in the 
hand of the user without the control element. 

Apart from the visualizing of virtual elements, such as 
coordinate systems or robot axes or also real elements in 
virtual form, such as a control unit on a manual program- 
mer, together with the image of the real robot according to 
a highly preferred development it is also possible to vi- 
sualize a robot and its movement sequence in a real envi- 
ronment and with the real workpieces present there. This 
reduces the risk of damage to things and injury to people. 
The time exposure is reduced, because the augmented robot 
can move at random speed, even if people are in its imme- 
diate vicinity. 

For teaching path points the simulated robot is moved by 
the user and the path points are stored on reaching the de- 
sired target position. When testing programs, the simula- 
ted robot covers the programmed path in place of the real 
system. This can take place at the programmed speed or any 
other random speed. The following of the path need not be 
simulated close to reality. It can be advantageous to me- 
rely successively fade in the target poses (i.e. to jump 
from path point to path point) , in order to check and op- 
tionally adapt them. 

According to a further development, a robot path to be tra- 
versed by a robot during an operation and in particular 
that of the tool centre point can be faded in together with 
additional information, preferably associated with the ro- 
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bot and more precisely the image of its real hand flange or 
the TCP of a tool. By programming or manipulating proces- 
ses, which are performed by an operator on a manual pro- 
grammer, there is a change in space to the display of the 
visualized robot path, which is displayed in association 
with the robot, i.e. extending through the hand flange or 
TCP of a tool. This can in particular take place in that 
path points are faded in, the position of the path points 
in space is variable and in each case a robot path curve 
corresponding to the position of all the path points is fa- 
ded in. 

In addition to the spatial path pattern, additional motion 
parameters (e.g. speeds, accelerations, etc.) or special 
functions (e.g. path switching functions, setting/reading 
of inputs /outputs, starting/stopping machining processes, 
etc.) can be visualized. This information can be visuali- 
zed in text form, but also graphically or by animation. 

According to a further development for adapting a robot 
path to the position of a determined real workpiece, a vir- 
tual image of the workpiece is faded in together with a ro- 
bot path adapted thereto, so that by superimposing the vir- 
tual workpiece image and the path points of the robot with 
the image of the real workpiece it is possible to adapt the 
robot path to be performed to the position of the real 
workpiece . 

According to other preferred developments of the invention 
the total operating area reachable by a robot and/or per- 
mitted operating areas and/or non-permitted operating areas 
are visualized on the display and/or movement corridors of 
a robot tool, robot hand and/or further robot elements are 
visualized on the display. "Total operating area" designa- 
tes the total number of all the poses which can be assumed 
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by the TCP. "Operating areas" can be fixed at random by 
the programmer . 

In the case of a plurality of robots, which e.g. interact 
in an installation, it is also possible to visualize perma- 
nent and/or instantaneous associations of at least one ma- 
nual programmer of at least one robot. 

In a similar manner to the association between the robot 
system and manual programmer, it is also possible to visua- 
lize and if necessary modify via the AR the present state 
of the robot system. Examples for states to be visualized 
are the selected operating mode, the state of the drives 
(on or off) , etc . 

It is common to all the described, preferred developments 
that the computer-generated informations can either be con- 
stantly visualized for the user or when requested by him 
(e.g. in conjunction with the manual displacement in the 
case of an operation of a displacement key if the authori- 
zation key is not pressed) . According to another alterna- 
tive the information is automatically faded in at a prede- 
termined time (e.g. shortly before the real robot system 
starts moving) . 

In the above-described four use sectors the AR system can 
supply a continuous image sequence or, initiated by the 
operator, individual "snapshots". Correspondingly the in- 
formation is faded in either online into the image sequence 
or offline into the recorded individual image. 

Visualization can also be switched off. In the case of a 
continuous image sequence it is optionally also possible to 
freeze the image. 
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The visualization of generated objects can have random de- 
grees of reality proximity. Visualization can also be per- 
formed in semi transparent manner or as an edge model, in 
order not to impair the viewing by the user of the plant. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Further advantages and features of the invention can be 
gathered from the claims, the following description of em- 
bodiments of the invention and the attached drawings, whe- 
rein show: 

Fig. la, lb A first embodiment of a device for fading 

computer-generated informations into an 
image of the real environment in plan view 
(fig. la) and in side view (fig. lb) . 

Fig. 2a, 2b Another embodiment of the inventive device 

in plan view (fig. 2a) and in side view 
(fig. 2b) . 

Fig. 3 Another embodiment of the inventive 

device . 

Fig. 4a-4c Representations regarding the manipulation 

of augmented objects spatial made possible 
by the invention in several variants . 

Fig. 5 In exemplified manner a flow diagram 

concerning the AR-assisted modification of 
a robot path. 



Fig. 6 



A block diagram of the device according to 
the invention. 
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Fig. 6a 



A block diagram of the device according to 
the invention corresponding to the 
embodiment of fig. 2a. 



Fig. 6b 



A block diagram of the device according to 
the invention corresponding to the 
embodiment according to fig. 3. 



Figs. 7a-c 



Representations of selectable coordinated 
systems and displacement indicators (fig. 
7a) for manual, cartesian movement 



Figs. 8 to 17 



Combined representations producible accor- 
ding to the invention of areas of the real 
environment together with robot-specific, 
computer-generated information as 
augmented reality displays. 



DETAILED DESCRIPTION OF THE DRAWINGS 

Figs, la and lb show a first embodiment of an inventive de- 
vice 1 for fading computer-generated inf ormations into an 
image of the real environment on a display. 

The device 1 according to the invention has a manual pro- 
grammer 2 for a robot with which the operating sequence of 
a robot can be comfortably programmed. The manual program- 
mer 2 is provided for this purpose with control or opera- 
ting elements in the form of six displacement keys 2.1 and 
a 6D mouse 2.2 by means of which the movements of the six 
axes of a robot can be controlled in the same way as 
through the keys 2.1. Such a manual programmer is known 
from EP 840 910 (corresponding to US 6,134,102) or EP 840 
9 09 (corresponding to US 6,3 62,813), to which explicit re- 
ference is made in connection with the disclosure of such a 



15 



manual programmer . The manual programmer 2 also has a 
screen 2.3 on which are displayed information and data of 
the control program, such as different parameters concer- 
ning the movement sequence for the robot movement. It is 
e.g. possible to input by means of not shown, physical keys 
or in that the screen 2.3 is constructed as a touch screen 
and input keys are faded onto the screen 2.3. 

In the embodiment according to figs, la and lb, separated 
from the screen 2.3, the inventive device has an additional 
viewing device 1.1, which nevertheless is physically con- 
nected to the manual programmer 2 and can e.g. be pivotably 
fitted thereon. In the represented embodiment on the back 
of the viewing device 1.1 is provided an image receiving 
device 1.2 in the form of a camera by means of which the 
real environment can be recorded. As a result of this re- 
cording, e.g. by markings positioned in space, it is possi- 
ble to mathematically determine the position and orientati- 
on in space, for short referred to as the pose of the image 
receiving device 1.2. Alternatively the pose of the image 
receiving device 1 . 2 can be provided by a comparison of the 
actual image recorded by it with a stored reference image 
or by using an additional sensor system. 

On the viewing device 1.1 is shown on the one hand the 
image of the real environment recorded by the camera 1 . 2 
and on the other robot-specific, computer-generated infor- 
mation is also faded into said image, as is e.g. shown in 
the views of the device 1.1 in figs. 7 to 17 . 

In place of a viewing device 1.1 in the form of a conven- 
tional graphicable screen, said device can also be a trans- 
parent screen, so that, optionally by means of an imaging 
optics can be shown and rendered visible on said viewing 
device 1.1 an image of the real environment without optoe- 
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lectronic conversion. The determinations of the pose can 
e.g. also take place technically in such cases, in that ra- 
dio markings are provided in space and relative to these it, 
is possible to determine the pose by means of a receiver 
which is at least connected to the viewing device 1.1. 

The viewing device 1.1 can also be spatially separated from 
the manual programmer 2, as is e.g. the case in the special 
construction according to fig. 3. 

In place of a separate screen as the viewing device 1.1 and 
as is e.g. the case in figs, la and lb, the viewing device 
1.1 showing an image of the real environment together with 
the robot-specific, computer-generated information can also 
be integrated into the manual programmer 2, e.g. as a 
screen window within the screen display 2.3 of the manual 
programmer 2, as is the case in figs. 2a and 2b. 

In such a case a camera 1.2 for recording the real environ- 
ment is positioned directly at the manual programmer 2, as 
is particularly apparent from fig. 2b. It can also be com- 
pletely separate from the viewing device 1.1, particularly 
if e.g. an appropriate recording location is not or is not 
readily accessible to the user. 

Otherwise the construction of figs. 2a and 2b corresponds 
to that of figs, la and lb, particularly with respect to 
the manual programmer 2 . 

Fig. 3 shows a device according to the invention with a 
viewing device 1.1 in the form of spectacles, where the 
user 4 can directly see the real environment through lenses 
1.3 (see-through spectacles), the lenses 1.3 being con- 
structed as a transparent screen into which can be faded 
the robot-specific, computer-generated information. For 
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this purpose the viewing device 1.1 is connected by means 
of a cable 1.4 to a manual programmer 2, which can also be 
replaced by wireless transmission, particularly radio 
transmission, but also infrared transmission. At the vie- 
wing device 1.1 is provided a camera 1.2 by means of which 
and using corresponding optical markings in space, it is 
possible to establish the pose of the viewing device 1.1. 
Additionally or alternatively the pose of the viewing devi- 
ce 1.1 or image receiving device 1 . 2 can be obtained by a 
comparison of the camera image with a stored image. It is 
fundamentally also possible to provide an electromagnetic, 
optical or acoustic sensor system by means of which it is 
possible to establish in the manner described hereinbefore 
the pose of the image receiving device or viewing device 
relative to corresponding marking transmitters in space. 

The invention permits a manipulation of robot-specific data 
by the manipulation of virtual or augmented robot-specific 
space objects, such as the robots themselves, path pattern, 
workpieces, etc. and this will be explained relative to 
figs. 4a to 4e. 

The known control elements on the manual programmer, such 
as the displacement keys/6D mouse can be used, by operation 
through the hand 4.1 of a user, for shifting and rotating 
the selected, augmented space objects, AR' indicating the 
pose of the shifted space object, similar to the TCP of a 
real robot system being moved in space. Compared with pu- 
rely AR visualization, the time involved is limited. 

The position of an input element in space can be determined 
by a sensor system 3 with a stationary receiving system 3.1 
and an input element 3.2, e.g. in the form of a pointer. 
If a user moves an input element 3.2, this movement is 
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transmitted to the selected, augmented space object AR (or 
optionally several of these) . 

In order to improve the ergonomy, the movement can be 
transmitted to the selected space object on an increased or 
decreased scale corresponding to a scaling factor preset by 
the user. The production of the "coupling" of the spatial 
movements of the input elements and the augmented space ob- 
ject takes place through a corresponding input on the part 
of the user (e.g. for as long as he keeps a button on the 
input element depressed) . By alternating "coupling" and 
"decoupling", it is also possible to displace over conside- 
rable distances augmented space objects without the input 
element having to be moved over a correspondingly large di- 
stance, so that an "indexing" takes place. 

Input elements in the aforementioned sense are e.g. a poin- 
ter, a handle, a data glove or the actual manual program- 
mer. The sensor system can be based on electromagnetic 
(e.g. radio), optical (e.g. infrared) or acoustic (e.g. ul- 
trasonic) tracking. If the manual programmer is used as 
the input element, the aforementioned system already pre- 
sent therein for determining the spatial pose can also be 
used for manipulating space objects. 

Alternatively the pose of the input element (e.g. a poin- 
ter) can be measured and evaluated relative to the hand- 
held manual programmer instead of the ground. In addition 
to the aforementioned possibilities, the pose of the input 
element can also be used through a mechanical measuring sy- 
stem between the input element and the manual programmer, 
such as multi joint kinematics with integrated joint angle 
sensors . 
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Space objects can be directly preset using an input element 
3.1 (e.g. a pointer, a handle, a data glove or the complete 
manual programmer) , whose pose in space is detected by a 
corresponding system . 

The essential advantage of the manipulation possibilities 
according to figs . 4b and 4c is that the manipulation of 
space objects via the spatial positioning of an input ele- 
ment saves time and is more intuitive for the user than 
when using a speed-controlled displacement and rotation, 
e.g. using movement keys or a 6D mouse. 

The described manipulation possibilities can be imaged on 
any of the described AR use senarios (shifting of the refe- 
rence coordinate system during the measurement of tool and 
workpiece coordinate systems, displacement of the support 
points of robot paths, displacement of the corner points or 
side walls of operating areas, etc.). 

The sequence for an AR-assisted modification of a robot 
path shown in fig. 5 initially involves a reprogram selec- 
tion with an interrogation of the machining status to esta- 
blish whether or not it has been concluded (step A) . If 
the answer is in the negative there is an interrogation as 
to whether a path section is to be modified (step B) . If 
this is not the case there is a return jump. If a path 
section is to be modified, then in the program there is a 
determination or calculation with respect to a selected 
path section and this can include the entire path of the 
tool centre point (step C) . The selected path section or 
the entire path are faded into an image of the real envi- 
ronment on the viewing device (step D) . This is followed 
by an interrogation to establish whether a displacement in- 
struction or several such instructions are to be modified. 
If this is not the case the visualization is ended (step 
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D') and the path section faded out (step D") and there is a 
return to interrogation B. 

On modifying a displacement instruction or several of these 
optionally the associated path is determined (step F) . The 
target of the selected displacement movement and optionally 
also the associated path of the tool centre point is 
graphically accentuated in the AR display (step G) . Optio- 
nally also the orientation is appropriately visualized by 
fading in a coordinate system, as will be explained herein- 
after relative to fig. 14. 

On the basis of the preset details described in connection 
with figs. 4a to 4c on the part of the user by the shifting 
of the path and optionally the target in space, now modi- 
fied targets and optionally paths are determined (step H) 
and visualized in the AR display (step I) , so that the po- 
sition of the target and optionally the path in the AR dis- 
play can be checked. Optionally the software of the robot 
control checks whether the target can be reached by the ro- 
bot and optionally generates warning messages. 

After calculating and displaying desired targets and optio- 
nally paths in accordance with steps H and I, there is an 
interrogation as to whether the desired target and optio- 
nally the desired path pattern has been reached (step J) . 
If this is not the case, the determination and visualizati- 
on of another target and optionally path pattern can be re- 
peated, so that steps H and I are. repeated. If the desired 
target has been reached, there is a securing thereof and 
the associated movement instruction (step K) . 

There is then a return to interrogation E as to whether 
another displacement instruction is to be modified. If 
this is the case steps F to K are performed again for the 
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same, whereas if it is not the case, as explained relative 
to steps E', E" , there is a return to the interrogation as 
to whether a further path section is to be modified (step 
B) . If this is not the case, as has been explained, there 
is a return to the interrogation concerning the machining 
status and, if no further machining is to take place, a de- 
selection of the program and therefore the ending thereof 
(step L) . 

The insertion of a further target of a displacement move- 
ment takes place correspondingly, a starting position being 
inputted or on the part of the program can be automatically 
proposed and the target position of the preceding movement 
can be present therein. 

The represented program sequence can be further automated 
and simplified, in that e.g. robot paths are automatically 
faded in directly after opening the program and/or a selec- 
tion of possible robot paths is automatically visualized. 
The user can optionally observe corresponding robot paths 
in augmented reality in a type of preview on marking a pro- 
gram in a program survey of the manual programmer. 

Figs. 6 to 6b show the structure of the inventive device 
for fading in robot-specific, computer-generated informati- 
on into an image of the real environment and the sequence 
according to the invention is readily apparent therefrom. 

The inventive device 1 has a processing unit 5 for proces- 
sing robot-specific, computer-generated information for the 
purpose of displaying the same on the viewing device 1.1. 
To the processing unit 5 can be connected a camera 1.2, as 
shown in broken line form in fig. 4, which records an image 
of the real environment and supplies it to the processing 
unit 5. Within the processing unit 5 there is an image ge- 
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nerating unit 5.1, with evaluates the pose of the viewing 
unit 1.1 in accordance with the camera 1.2 and mixes the 
camera image with the robot-specific, computer-generated 
information, so that the real image and the information to 
be displayed are jointly displayed on the viewing* device 
1.1. 

For this purpose the processing unit 5 has a manipulating 
unit 5.2, a robot control 6 having data interfaces for con- 
trolling at least one robot and obtaining therefrom robot- 
specific information, which together with the image of the 
real environment are displayed on the viewing device 1.1. 
The processing unit 5 can also incorporate an interface for 
input devices, such as e.g. a manual programmer for a ro- 
bot, which allow a spatial manipulation of the robot- 
specific information by means of a human user. By means of 
the device according to the invention, robot-specific in- 
formation, optionally whilst taking account of user inputs, 
are used in augmented reality models, which are in turn 
further processed by the image generating unit 5.1 for 
displaying the augmented image on the viewing device 1.1. 
Changing robot data, optionally due to user inputs, can be 
returned by the model manipulating unit 5.2 to the robot 
control 6. A user input for the spatial manipulation of AR 
data is designated 5.3. 

The processing unit 5 can be integrated into either the 
viewing device 1.1, a manual programmer 2 or the robot con- 
trol 6. It can fundamentally also form its own physical 
unit. The processing unit 5 can also be spatially subdivi- 
ded, e.g. in that the image generating unit 5.1 is located 
in the viewing device 1 . 1 or a manual programmer 2, whereas 
the model manipulating unit 5.2 can be housed in the robot 
control 6. The processing unit 5 or parts thereof can be 
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combined into a central system for different robots and op- 
tionally communication paths can pass via the Internet. 

Fig. 6 shows a basic case and from it are derived in figs. 
6a and 6b special cases for hand-held or spectacle-like 
viewing devices. The general architecture represents the 
points in common between the two specific architectures 
described hereinafter. 

In the construction according to fig. 6a the viewing device 
1.1 is connected to or integrated into a manual programmer 
2 in accordance with figs, la, lb, 2a and 2b. It is provi- 
ded with a camera 1.2 and input elements 2.1, 2.2 (keys, 6D 
mouse). The camera 1.2 is able to determine its position 
and therefore that of the viewing device 1.1 in the manner 
described, whilst by means of the input elements 2.1, 2.2 
inputs for manipulating robot data can be inputted by the 
human user. Otherwise the construction according to fig. 5 
corresponds to that of fig. 4. 

Fig. 6 shows the block diagram concerning the inventive de- 
vice of fig. 2. The viewing device 1.1 is integrated into 
augmented reality spectacles, which can also be equipped 
with a camera 1.2. The spectacles can be of the see- 
through type, in which the real world is perceived directly 
and not via a camera image and only the computer-generated, 
information is faded in. Apart from a spatial input ele- 
ment, e.g. a manual programmer 2 with input keys 2.1 and/ or 
a 6D mouse 2.2, a spatial viewing device 1.1 is provided in 
physically separated form. Otherwise the block diagram of 
fig. 6 corresponds to that of fig. 4. 

The input device in the form of a manual programmer 2, also 
to the extent that it acts as a viewing device 1.1, can al- 
so be provided in a different construction, e.g. a diffe- 
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rent input device and a different viewing device can be 
provided and in particular the latter was described in this 
connection hereinbefore. 

Figs. 7a to 7c show possible representations on the viewing 
device 1.1 with areas of the real environment, such as in 
particular a robot 7 or a workpiece 8 being machined by it, 
as well as robot-specific, computer-generated information 
faded into this image of the real environment and connected 
thereto. It is possible with the image of a real robot 7 
with a tool 7.1, such as welding tongs, to generate coordi- 
nate systems to be associated therewith, such as the sta- 
tionary world coordinate system Kl with the coordinates X+, 
Y+, Z+. The coordinate system K+ is a world coordinate sy- 
stem displaced in parallel into the TCP with displaced 
coordinates X'+, Y'+, Z' + . The origin of the world coordi- 
nate system Kl is at the intersection of the axis of symme- 
try of the base 7a of the robot 7 with the background and 
the origin of the coordinate system K2 is in the centre of 
the hand flange or, when a tool is present, is displaced 
into the tool centre point. Through the representation of 
coordinate systems, assistance is obtained in manual carte- 
sian movements of the robot, particularly by means of the 
displacement key on the programmer (fig. 7a) . 

The black arrow V in fig. 7a by its direction and length 
visualizes the movement direction and speed with which the 
robot was moved if it was ready to move, for as long as it 
is not ready to move (drives not on, permission key not de- 
pressed) . If the robot moves, the arrow V visualizes the 
actual movement direction and speed of the robot system. 

, In order to facilitate the drawing up, adaptation or te- 
sting of robot programs, the workpiece 8 to be machined can 
be visualized in computer-generated manner in association 
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with the image of a real robot 7. The simulated component 
8' then forms the reference during the drawing up, adapta- 
tion and testing of the robot program. The use of a simu- 
lated component 8' can also be advantageous if the real 
component 8 is bulky or hinders the free movement of the 
user 4 in the plant (fig. 7b) . 

Fig. 7c shows the fading in of a tool coordinate system K3 
with the coordinates X 3 +, Y 3 +, Z 3 + as AR assistance in manu- 
al cartesian movement. 

The tool or workpiece can be measured by manipulating the 
reference coordinate system. The sequence is similar to 
the manipulation of targets of path movements, as described 
relative to fig. 5. The reference coordinate system is 
shifted and rotated in space under the visual control of 
the user until it is in the desired position relative to 
the workpiece (when measuring a basic coordinate system) or 
the tool (when measuring a tool coordinate system) . 

If a robot path is to be programmed or modified and no real 
workpiece is available, according to the invention in place 
of this it is possible to operate with an augmented work- 
piece. A user teaches the points by manual movement of the 
robot on the augmented workpiece and not on the real work- 
piece. In the same way as when a workpiece is present, it 
is also possible to augment absent plant components in or- 
der to e.g. fix the robot path in such a way that following 
installation of said plant components no collisions occur. 
The manual movement of the robot in a space into which are 
faded (almost) only augmented objects is simpler, because 
there is no need to take account of robot collisions with 
real objects and the user can move unhindered in space. 
The augmented objects are here of a static nature. 
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If no real workpiece is present the visualization of the 
workpiece by AR can possibly provide valuable additional 
information to the user as to which movement instructions 
of the robot belong to a specific machining process (in 
that possibly very complex robot paths are visualized, so 
that a survey is difficult) . 

The visualization of the augmented workpiece reveals to the 
user the workpiece from which the robot is starting. If a 
robot machines several workpieces with different programs, 
the comparison between the real and augmented workpiece 
provides the user with a simple possibility of veryfying 
the choice of the correct machining program (in this con- 
nection see the description of fig. 13). 

The augmented workpiece indicates to the user at which 
point the robot assumes the workpiece to be. If the robot 
program path does not correspond to the position of the re- 
al workpiece (e.g. due to positional tolerances), the user 
can correct the robot program in that he shifts and rotates 
the augmented workpiece together with the associated robot 
program path points until coincidence with the real work- 
piece is obtained. The manipulation can take place in the 
manner described hereinbefore. 

In the last three application cases described the visuali- 
zation of the workpiece by AR represents additional infor- 
mation for augmented path patterns of the robot program. 
The augmentation of the workpiece is dependent on the robot 
path (its position changes e.g. due to the shifting of the 
corresponding workpiece coordinate system) . Consequently 
this represents dynamic, manipulatable, robot-specific in- 
formation . 
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For assistance in the case of a manual axis-specific move- 
ment of a robot, particularly with the movement keys on the 
programmer, additionally and instead of a coordinate system 
the axes Al to A6 of the robot can be displayed, i.e. they 
are so faded into the image of the real robot 7 that they 
coincide with its actual axes in the image. In addition, 
positive and negative rotation directions about the cor- 
responding axes can be displayed (fig. 8) . The visualiza- 
tion of the axes can also be helpful for manual cartesian 
movements in order to indicate to the user in the vicinity 
of singular joint configurations that one or more axes will 
soon arrive at the limits of their maximum acceleration or 
speed. 

In addition, manual movement more particularly by means of 
the 3D mouse 2.2 can be facilitated in that this is faded 
directly upstream of the hand flange 7b or TCP of a robot 7 
into the image thereof as a virtual 3D mouse 2.2'. It is 
additionally possible to also fade in the virtual hand 4.1a 
of a user (fig. 9 ) . 

Fig. 10 shows the image of a real robot 7 together with 
computer-generated, further positions 7' and parts of the 
robot, namely rocker, robot arm and robot hand, in diffe- 
rent positions, so that total operating areas and the like 
are illustrated and therefore programming of an operating 
sequence of the robot can be facilitated, because for the 
teaching of path points and the checking of a programmed 
path of the robot there is no need to move, so that time is 
saved and the danger of injury and damage is reduced. 

It is also possible, together with the image of a real ro- 
bot 7 and passing through its hand flange 7b or TCP, to fa- 
de in a computer-generated robot operating path, optionally 
together with specific, alphanumerically designated points 
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Pi to P6 of the path and path speeds in certain areas, such 
as areas between two points and finally also alphanumeric, 
functional details, such as in the embodiment shown the 
opening and closing of a gripper (open gripper, close grip- 
per) at the path points P4 and P5 (fig. 11) . 

In addition to the alphanumerical informations, it is also 
possible to visualize symbols or the orientation of the 
tool in specific path points. 

For the adaptation of existing robot programs the path 
points can be directly, individually or groupwise displaced 
and rotated in space, in the manner shown in fig. 12, the 
provision of new path points or also complete new program 
or path sequences also being directly visualizable, as ex- 
plained relative to fig. 5. 

According to a further development, together with the image 
of the real component 8 a computer-generated component 8 ' 
can be visualized (fig. 13) . Thus, in the case of diver- 
gences in the component position, the position of groups of 
path points in the robot program can be corrected, in that 
the augmented workpiece 8 ' together with path points or the 
path of the robot relating thereto can be displaced and ro- 
tated until the augmented workpiece 8 ■ coincides with the 
real workpiece 8 . 

It is also possible to visualize and interactively adapt 
operating areas of a real plant comprising several robots. 
Fig. 14 shows the real image of three cooperating robots 
7.1, 7.2, 7.3 together with the computer-generated repre- 
sentation of their total operating areas 7.1', 7.2', 7.3'. 
The visualization of the attainable total operating area of 
a robot system can be advantageous for detecting possible 
collision zones of closely juxtaposed robot systems. The 
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direct application or adaptation of the robot path is con- 
sequently effectively assisted by the visualization of the 
attainable operating area. 

It is also possible to visualize and adapt areas which the 
robot must not penetrate or must not leave. These can also 
be operating areas which are jointly used by several robot 
systems or other plant components, so that their use can be 
managed. Virtual fixtures, which are used in the manually 
guided operation of robot systems can be easily fixed, vi- 
sualized and adapted. The types of operating areas can in 
each case be defined by the programmer or automatically fi- 
xed by corresponding software. The defined operating area 
can in each case relate to a defined point of the robot 
structure, preferably the TCP or alternatively the outer 
contour of the robot or part thereof. 

The visualization of an operating area can e.g. take place 
by fading in a body of the same size and geometry cor- 
responding to the operating area (fig. 15) . The visualiza- 
tion of virtual fixtures or guidance corridors can take 
place in a similar way and use can be made of three, two or 
one-dimensional bodies, as shown in fig. 16. It can alter- 
natively be advantageous to visualize an equivalent mecha- 
nical structure, which incorporates a corresponding re- 
striction of the movement possibilities, such as the visua- 
lization of guide rails, joint mechanisms, etc. 

To adapt visualized operating areas it is possible to indi- 
vidually correct in space corner or auxiliary points. It 
is alternatively possible to adapt operating areas in that 
their lateral faces are individually displaced and rotated, 
as described relative to figs. 4a to 4c. 



30 



The inventive AR visualization of operating areas and vir- 
tual fixtures assists the user in fixing the same, e.g. by 
the fading in of permitted and non-permitted operating vo- 
lumes, as shown in figs. 15 and 16. During programming a 
possibility exists for manipulating the operating areas or 
virtual fixtures, e.g. via the "shifting" of corner points 
with the input elements on the manual programmer. 

If several robots can be successively programmed with a 
single manual programmer, it is possible to mark the pro- 
grammer-associated robots, e.g. by a graphic element such 
as an arrow or the like (fig. 17) . This is also the situa- 
tion if different robots can be successively or simultane- 
ously controlled by means of a manual programmer. Here 
again it is advantageous to make visually apparent to the 
user the robot presently associated with the manual pro- 
grammer. It is additionally possible to select and dese- 
lect robot systems by the user contacting faded-in, virtual 
input elements or the programmer is oriented with the cor- 
responding robot. The basic selectability of a robot or 
its lack of usability e.g. if the robot does not belong to 
the group of robots which can be controlled by a given ma- 
nual programmer, can be indicated directly to the user on 
the image of the real robot by a corresponding marking or 
designation. It is also possible to visualize and if ne- 
cessary modify the actual state of robot systems, such as 
the selected operating mode, state of the drives (on, off) 
or the like. 



